1. Field of the Invention
The present invention pertains to the field of pattern recognition. More particularly, the invention pertains to the field of circuits and methods for use in adaptive pattern recognizers capable of supervised or unsupervised memory modification or learning.
2. Description of the Related Art
Artificial neural networks (ANN's) having parallel, distributed processing are recognized as a way to provide extremely fast overall processing with relatively slow elements, pattern recognition by learning from prototypes with subsequent minimal functional degradation due to noisy inputs, and effective functioning despite deficient or inoperable elements, this last feature providing high yields of ANN integrated circuits and continued effective operation despite element failure and physical damage. Although extensive theoretical research and digital simulations have been performed in connection with ANN's, their practical application depends on the development of circuits and methods for learning or memory modification in an ANN embodied in a practical and mass producible integrated circuit having vast numbers of elements functioning in parallel.
A representative ANN is depicted conceptually and conventionally in FIG. 1, and has two "neurons" 10, each providing an output 11 of the network, and four data inputs 12. The inputs are individually connected to the neurons by eight "synapses" 13. To make exposition possible, the network is much simpler than any practically useful ANN integrated circuit having, for example, sixty-four neurons each with one hundred and twenty-eight inputs. Typically, an artificial neural network, such as that of FIG. 1 but having many more elements, is used as one "layer" of a more complex network in which outputs, such as 11, of one layer are data inputs, such as inputs 12, to another layer. It is evident that such complex networks are best practically implemented with very large scale integrated circuits and, typically, with complementary metal oxide semiconductor (CMOS) technology. Therefore, methods and circuits described herein, both of the prior art and of the present invention, should be considered as they might be implemented in very large scale integrated circuits in which several connected such layers each have thousands of synapses.
In FIG. 1 each synapse 13 is depicted as having a variable conductance 15, represented by the conventional symbol for a variable resistance to indicate that a signal, typically a voltage signal, on each input 12 determines a signal, typically a current signal, to a summing circuit 16 of each neuron 10. The variable conductance may be provided by an active circuit, and the synapse may function as a sink and/or a source of the current determined by the synapse. This current is proportional to the strength of the corresponding input signal and to a "weight" which is the conductance 15 of the synapse. The output of each circuit 16 corresponds to the sum of the currents determined by the associated synapses and is provided to a function generator 18 which drives the corresponding output 11 to provide a, typically voltage, output signal. This output signal thus corresponds to a predetermined relation or activation function applied to such sum of the currents. It is well-known that by suitable adjustments to the weights, that is, modification of the "memory" of the network occurring when the memory "learns", the output signals may be made to assume desired values corresponding to predetermined values of the input signals. The usefulness of an artificial neural network is thus dependent on the practicality of its arrangements for memory modification as embodied in an actual circuit.
Several kinds of circuit elements may be used to provide the synapse weights or conductances 15 and for adjusting the weights, so that the particular circuit arrangements employed in an artificial neural network using a method of the present invention depend on many considerations including the manner of implementing the variable weights. The variable weights may be implemented by an actual variable resistance of some kind, by a digitally switched resistance network, or by a multiplying analog to digital converter (MDAC) in which a variable, stored digital value is, in effect, multiplied times an input signal to generate an analog signal representing the product of the digital value and a value represented by the input signal current. Each variable synapse weight may also be controlled by a charge on the gate of a field effect transistor, and the use of a floating gate field effect transistor with the transistor serving as a memory element for a weight value represented by the gate charge is believed highly advantageous in an ANN.
In memory modification of an ANN, the general approach is to provide the inputs 12 with successive patterns and, at each presentation, to determine the errors between the values of the signals on the ANN outputs 11 and their desired values. The errors are then used in accordance with the method to modify the weights. In order to carry out memory modification, the ANN of FIG. 1 has elements which are conceptually depicted. These elements include a neuron or column select input 20 to a multiplexer 21 from which two select signal conductors 22 extend individually through the neurons 10 and has an input or row select input 25 to a multiplexer 26 from which four select signal conductors 27 extend individually through the four synapses 13 associated with each input 12. Each synapse 13 has an AND gate 31 which is connected to the corresponding conductors 22 and 27 and which outputs a signal when the synapse is selected by addressing through inputs 20 and 25. Each synapse is depicted as having a weight setting circuit 33 which is activated when the synapse is selected. The ANN has a weight adjustment input 35 from which a conductor 36 branches through the ANN to the setting circuit 33 of each synapse 13. Each synapse is represented as having a switch 37 which is closed to permit setting circuit 33 to adjust conductance 15 of the synapse. Each switch 37 is controlled by a signal provided on a conductor 38 from an input 39 of the ANN. Setting circuit 33 has any suitable structure, which depends on the manner of implementing the conductances 15 and the memory modification method to be used with the ANN, for adjusting the synapse weight represented by a conductance 15 to a desired value determined by a signal provided at input 35. The weight adjustment signal may be a digital value or an analog value representing the adjusted weight, may be a digital or an analog value representing a desired change in the weight, or may be a relatively high voltage pulse for directly modifying a charge on a floating gate.
A well-known memory modification method proposed for an ANN has been described in connection with a multiple adaptive linear neuron or "Madeline" and is commonly referred to as the "Madeline method". In the Madeline method the function provided by each generator 18 is a "signum" or step function so that the output of each neuron 10 can only be in one of two states. In each cycle of learning by the Madeline method, it is necessary to determine the one of the neurons 10 having the input to the function generator 18 of the neuron from the corresponding summing circuit 16 such that this neuron is the one in the network whose output is nearest to changing between the states. The synapse weights or conductances 15 of this neuron are then adjusted and the cycle is repeated. In each memory modification cycle of the Madeline and related methods, the currents to the summing circuits 16, which represent the sums of the signals on the inputs 12 weighted by the corresponding conductances 15, may first be perturbed successively to determine such one neuron nearest to changing. Next, the overall error of the network is determined before and after such a perturbation in current to the summing circuit of the one neuron. Third, if this overall error is decreased by such a perturbation, the conductances 15 of the synapses 13 of the one neuron are adjusted by an amount which is proportional to the perturbation, to the reduction in error, and to the individual signals to the inputs 12 of these synapses. The cycle is repeated until the conductances 15 converge to values providing desired values for the outputs 11.
However, the present invention is not limited to use with the Madeline memory modification method just described, and may be used with other methods wherein a function generator such as generator 18 provides, instead of a signum function, any other suitable function such as a "sigmoid" or S-shaped function indicated in FIG. 1. The present invention is effective when used with such another memory modification method where the neurons are perturbed sequentially, this latter method being generally that of U.S. Pat. No. 5,075,686 issued Dec. 24, 1991 and being subsequently set forth below for illustrative purposes in connection with the subject application.
The ANN of FIG. 1 has conceptually represented elements for carrying out such perturbation in accordance with the prior art by injecting a predetermined perturbation current into the summing circuit 16 of a selected neuron. These elements include a perturbation current input 40 receiving the current, a perturbation current conductor 41 extending through the neurons 10 and branching to each summing circuit, and a switch 44 for each neuron, switch 44 being closed by a signal from an AND gate 46 when the neuron is selected by signals provided to gate 46 on the corresponding neuron select conductor 22 and on a perturbation select signal conductor 47 branching through the ANN from a perturbation select input 48. Perturbation thus occurs when a neuron is selected for perturbation by its gate 46 current is being injected through input 40.
FIG. 2 is a diagram showing details of representative prior art circuits used together in a single neuron 10 and a single synapse 13 of a prior art ANN such as that of FIG. 1. Similar circuits effective in an ANN embodying the present invention will be subsequently described. The FIG. 2 circuits are well-known and are like those of FIG. 1 in being associated with an output 11, a data input 12, column and row select conductors 22 and 27, and elements 31, 33, and 36-38 for weight adjustment. However, the circuits of FIG. 2, which are preferably implemented with CMOS technology, are characterized by synapse 13 having a current sink arrangement serving as a two-quadrant multiplier 50 and including a floating gate FET 51, by neuron 10 having a positive summing node conductor 55 and a negative summing node conductor 56 extending through synapse 13, and by neuron 10 having a transconductance amplifier 58 connected to conductors 55 and 56 and a transimpedance amplifier 59 driving output 11. Amplifiers 58 and 59 together perform the functions of the FIG. 1 summing circuit 16 and function generator 18. Each node 55 or 56 is supplied with current through a series resistor and diode arrangement 60, and the multiplier 50 of each synapse 13 is a drain for this current. Multiplier 50 has a pair of FET's 63 and 64 connected respectively to nodes 55 and 56, and the gates of these FET's are each connected at a common point 65 to the input 12 associated with the synapse. The floating gate of FET 51 stores a charge representing the above-described synapse weight and the channel of FET 51 is connected between FET 63 and ground so that FET's 51 and 63 are a current drain for node 55 with the drained current being proportional to the product of the signal on input 11 and this weight. The floating gate of FET 51 thus retains such weight as a factor, and FET's 51, 63, and 64 are a two-quadrant multiplier determining the effect of a input signal, as on such an input 12, on an output signal, such as that on an output 11, so that this effect is the product of such a retained factor and the level of the input signal. FET 64 is connected directly to ground and thus drains from node 56 a reference current proportional to the input 11 signal. FET 51 is represented as having a control gate 66 connected to switch 37 and setting circuit 33 for varying, in any suitable manner as before stated, the charge on the floating gate of a FET 51 in response to signals on conductors 36 and 38 addressing the synapse having this floating gate.
It is apparent to one skilled in the art that the elements just described may be selected so as to cause the differential voltage between nodes 55 and 56 to represent the sum of the products of the signals on the inputs, such as input 12, with the weights provided by the synapses, such as 13, corresponding individually to the inputs. This differential voltage is provided to well-known amplifier 58 at the gates of a pair of FET's 70 connected to ground through a bias FET 71. FET's 70 are connected in opposite legs of a current mirror 73 so that amplifier 58 has an output node 75 having a current determined by this differential voltage, this current having a sigmoidal relation to the differential voltage and thereby defining the above-mentioned activation function applied to the sum of the currents determined by each synapse. Output 75 is provided to any suitable circuit, such as amplifier 59, which converts the current at this node to a voltage at the corresponding neuron output 11 and directly proportional to the current determined by the sigmoidal relation.
Although ANN memory modification using perturbation of individual neurons as described above has been extensively studied by digital computer simulation and the construction of ANN's having a relatively limited number of neurons and synapses, a number of problems existing in the implementation of ANN memory modification in practically useful ANN circuits, particularly for ANN's embodied in very large scale integrated circuits, have not been addressed by the prior art.
One such problem arises from the perturbation of individual neurons, exemplified by a neuron 10 in FIG. 1, by injecting a predetermined current into the summing circuit 16 of the neuron from a single current source, whether located on or off an integrated circuit chip embodying the ANN. Such injection requires suitable multiplexing, conductors such as conductor 41, and switches such as switches 44 for what are relatively large currents, the conductors and switches being intertwined with other elements of the chip which, typically, are adapted to voltage signals and minuscule currents. While, the currents required for summing of weighted signals as described above may be relatively large, they are each confined within one neuron and need not be switched.
Another problem arises from the tolerances of integrated circuit construction which result in different conductances and other characteristics in what are intended to be substantially identical elements. As a result, supposedly identical perturbations of different neurons, such as neurons 10, of an ANN result in uncorrelated changes of the neuron outputs 11 so that the calculations, such as those described above, for changes in the values of the conductances, such as 15, do not result in convergence to the desired values of outputs 11 or, at least, require an excessive number of cycles for convergence.
Still another problem occurs when a conductance 15, which is determined by the charge on a floating gate of a field effect transistor, is repeatedly varied to change a neuron 10 between a perturbed state and an unperturbed state, this change in state being generated by changing between a floating gate charge giving a relatively high conductance and such a charge giving a minimum conductance. Changing the charge of a floating gate field effect transistor involves injection of electrons or holes through isolating material surrounding the floating gate with resulting damage to the isolating material. Therefore, loss of isolation of the floating gate is likely to occur before practical memory modification can be achieved by repeated perturbations generated by relatively large changes in a floating gate charge.